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ABSTRACT:  A.?  in  the  commercial  world,  modeling  and  simulation  (M&S)  is  increasingly  pervasive 
throughout  the  Navy.  There  is  a  constant  drive  to  expand  the  utility  of  these  M&S  through  distributed  M&S, 
federations,  and  other  combinatorial  simulations  in  the  High  Level  Architecture  (HLA),  FORCEnet,  and  other 
such  programs.  The  DOD  and  commercial  sectors’  increased  reliance  on  M&S  has  necessitated  a  method  to 
provide  documented  proof  of  M&S’  credibility.  Although  the  Navy  has  mandated  M&S  Verification,  Validation, 
and  Accreditation  (W&A)  to  provide  this  credibility,  no  single  standard  exists  to  execute  the  documentation 
process. 

Clearly,  this  lack  of  coordination  and  consistency  begs  the  development  of  a  single,  clear,  standard  method  and 
repository  to  facilitate  the  execution  of  this  mandate.  The  development  of  Department  of  the  Navy  (DON)  W&A 
Documentation  Tool  (VDT)  and  its  W&A  Markup  Language  (WML)  seeks  to  fill  this  need  by  standardizing  and 
streamlining  the  M&S  VV&A  documentation  process.  The  web-enabled  VDT  can  not  only  house  all  W&A 
documentation  and  its  associated  information  in  one  repository,  but  can  also  standardize  and  expedite  the 
documentation  process  through  intuitive  templates  and  inherent  flexibility  via  its  WML  base.  This  paper  builds 
the  case  for  a  W&A  documentation  standard  and  details  the  role  that  WML  and  the  VDT  play  in  the 
realization  of  standardization. 


1.  Introduction 

The  ever-increasing  use  of  M&S  across  all  areas  of 
global  communication,  education,  business,  and  day- 
to-day  functions  has  come  to  define  the  twenty-first 
century.  Seemingly  all  areas  of  everyday  life  has  come 
acquisitions  to  assessment  and  management,  weapon 
systems  testing  in  lieu  of  live  firing,  and  ship  and 
aircraft  design.  As  this  M&S  presence  continues  to 
evolve  into  reuse,  greater  federations,  and 
combinatorial  simulations,  the  need  for  a  standardized 
means  to  prove  the  credibility  of  these  M&S  becomes 
more  urgent. 


to  rely  on  M&S  to  enrich  our  world  with  greater 
capabilities  at  lower  costs.  It  is  not  surprising  then  that 
the  US  Navy  has  led  the  way  in  incorporating  this 
highly  malleable  technology  to  build,  assess,  and  meet 
their  technological  advancement:  from  training  and 
education  to  operations  and  experimentations, 

1.1  Background:  Verification,  Validation,  and 
Accreditation  (VV&A) 

The  DON  has  issued  a  mandate  for  M&S  VV&A 
(SECNAVINST  5200.40  [1]  and  5200.40A  draft  [2])  to 
address  this  need.  The  key  focuses  of  Naval  M&S 
VV&A  are  two-fold:  1)  to  ensure  the  M&S 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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developers  are  producing  credible  models  that  fulfill 
their  intended  uses  and  2)  to  present  accurate  and 
correct  information  to  the  users  so  that  they  can  make 
conscientious  and  informed  decisions  regarding  their 
use.  De  facto  accreditation  has  often  been  used  due  to 
the  lack  of  V&V  documentation.  Because  of  this  lack 
of  a  formalized  VV&A  standard,  the  credibility  of  the 
M&S’s  pedigree  is  generally  unsubstantiated. 
However,  as  M&S  is  increasingly  relied  upon  to  make 
realistic  representations  of  the  “real  world”  as  well  as 
critical  decisions  with  substantial  risks,  it  is  clear  that 
“it  has  worked  so  far”  no  longer  suffices.  With  such 
factors  as  the  safety  of  human  lives  and  the  reduction 
of  cost  and  time  dependent  upon  the  accuracy  of 
M&S,  it  is  clear  that  VV&A  must  be  accurate  and 
consistent. 

The  need  for  standardization  of  VV&A  is  thus  a 
formalized  set  of  processes  designed  to  ensure  the 
safety,  cost  effectiveness,  and  reliability  of  M&S.  To 
facilitate  the  execution  of  these  processes,  a  VV&A 
framework  and  processes  have  been  outlined  and  in 
the  Department  of  the  Navy  Modeling  and  Simulation 
Verification,  Validation,  and  Accreditation 
Implementation  Handbook  (2001)  [3]  and  The 

Modeling  and  Simulation  Verification,  Validation,  and 
Accreditation  Implementation  Handbook,  Volume  I: 
VV&A  Framework  [4].  Thus,  the  vehicle  for  creating 
M&S  credibility  is  set  into  motion  through  VV&A. 

1.2  Necessity:  Standardized  VV&A  Documentation 

Although  VV&A  creates  the  framework  and  processes 
required  to  ensure  the  credibility  of  M&S,  VV&A 
may  be  considered  an  incomplete  effort  without 
structured  documentation.  VV&A  documentation 
must  address  the  concerns  of  the  M&S  user  by 
demonstrating  its  credibility  with  solid  evidence.  In 
other  words,  VV&A  documentation  must  build  on  the 
M&S  requirements,  design  and  development,  and 
reporting  and  documentation  of  the  M&S 
performances’  end  results.  The  efficacy  of  VV&A 
hinges  upon  its  thoroughness  and  consistency. 

The  simplest  way  to  ensure  this  pedigree  of  VV&A  is 
to  standardize  the  documentation  process.  The  Navy 
Modeling  and  Simulation  Master  Plan  includes 
standards  as  one  of  the  enablers  to  “promote  the 
development  and  use  of  standards,  in  order  to  reduce 
M&S  production  costs,  while  enabling  consistent  and 
comprehensive  representation”  [5],  As  mentioned 
above,  although  the  need  for  standardization, 
facilitation,  and  expedition  has  been  identified,  no  one 
tool,  template,  or  language  exists  to  facilitate  such 
standardization.  Therefore,  it  is  not  surprising  that  the 
VV&A  documentation  process  remains  daunting, 
inefficient,  and  generally  unpopular.  By  extension,  the 
potential  for  legacy  M&S  reuse  under  such  conditions 
has  been  heretofore  hindered  by  this  lack  of  consistent 
documentation  and  the  arduous  amounts  of  research 
needed  to  assess  the  applicability  of  an  M&S  in 
question  for  the  new  user’s  purpose. 


Credible  and  consistent  VV&A  documentation  has 
remained  elusive.  Since  hesitation  to  conduct  VV&A 
continues  despite  a  SECNAVINST  mandate  and 
practitioners’  guides,  something  further  is  still  needed 
to  activate  and  facilitate  the  VV&A  documentation 
process.  In  an  effort  to  capture  the  problem,  the  Navy 
has  been  collecting  data  on  VV&A  implementation 
needs  for  the  last  three  to  four  years.  The  dominant 
stumbling  block  has  been  the  facilitation  of  its 
execution.  VV&A  agents  have  consistently  indicated 
that  the  most  difficult  aspect  of  implementation  was 
the  initiation:  when  and  where  to  start,  what  to 
consider,  and  how  to  bring  about  structured 
management  to  produce  precise,  complete,  and  useful 
documentation  that  users  can  use  to  make  informed 
decisions.  The  fundamental  problem  is  therefore 
clear:  implementation  guidance  alone  cannot  meet  the 
needs  of  consistency  and  reuse.  A  concrete,  tangible 
standard  for  reporting  information  that  is  easy  to  use  is 
needed. 

1.3  The  Utility  of  WML  and  the  VDT  in  VV&A 

The  clamor  to  simplify  the  documentation  process 
with  an  intuitive  format  is  the  foundation  for  the 
creation  of  WML.  The  evolution  of  the  Navy’s 
effort  to  support  VV&A  documentation  has  brought 
several  key,  M&S-wide  desideratum  to  light:  the 
possibility  of  VV&A  standardization;  the  need  to 
promote  M&S  reuse,  and  the  need  for  easy,  on-line 
VV&A  collaboration.  Richly  structured  VV&A 
processes  are  possible  by  utilizing  the  Extensible 
Markup  Language  (XML)  to  create  universal  access  to 
M&S  data.  The  WML  concept  directly  addresses 
these  VV&A  needs  by  leveraging  the  XML-based 
framework  into  a  VV&A-specific  language,  thereby 
providing  a  bridge  between  M&S  requirements  and 
open/commercial  web-enabled  standards.  Further,  the 
execution  of  WML  in  the  VDT,  with  its  intuitive 
templates  and  user  interface,  solves  both  the  initial 
documentation  and  reuse  standardization  problems. 
The  associated  benefits  of  such  a  marriage  are 
numerous:  adaptability  to  changes  in  VV&A  structure 
and  data,  portability  to  enable  universal,  web-enabled 
access  to  VV&A  information;  the  capacity  to  link  to 
the  Navy  Modeling  and  Simulation  Resource 
Repository  (NMSRR)  [6];  and  easy  maintenance  and 
reference. 

2.  Purpose: 

To  date,  no  standards  specify  what  information, 
documentation,  or  format  is  required  for  M&S 
VV&A.  However,  observation  the  common  needs  of 
numerous  M&S  VV&A  programs  reveals  that  an 
efficacious  and  accommodating  VV&A 
documentation  process  hinges  on  two  key  factors:  a 
common  documentation  standard  and  a  method  by 
which  to  achieve  this  documentation.  This  paper  will 
show  that  the  WML  provides  the  extensible  language 


necessary  to  generate  a  common  VV&A  template  and 
that  VDT  provides  a  standard  content  and  format  for 
all  VV&A  implementation.  Continued  discussions, 
collaboration,  and  incorporation  of  M&S  VV&A 
needs  will  only  provide  ever  greater  utility  of  the 
WML  and  VDT. 

Thus,  this  paper  will  argue  the  need  and  justification 
for  a  single  M&S  VV&A  documentation  standard; 
provide  a  walkthrough  of  the  WML  concept  and  its 
application  through  the  VDT;  and  delineate  the 
evidence  needed  to  place  WML  and  the  VDT  as 
considerations  for  M&S  VV&A  standards.  By 
extension,  the  need  for  WML  and  the  VDT  in  the 
context  of  the  Navy’s  emphasis  on  M&S  reuse  and 
future  endeavors  will  be  highlighted.  WML  and  the 
VDT  will  prove  to  be  inextricably  vital  to  the  creation 
of  a  formal  and  consistent  means  to  evaluate  M&S 
and  the  creation  of  federative,  distributive.  Defense 
Information  Infrastructure  Common  Operation 
Environmental  (DII  COE),  and  composable  M&S. 

This  argument  will  be  laid  out  in  three  parts:  the 
presentation  of  the  WML  concept,  its  utility  as 
facilitated  by  the  VDT,  and  the  justification  for  its 
consideration  as  the  foundation  for  future 
standardization.  Finally,  the  discussion  will  conclude 
with  future  VVML/VDT  capabilities  and  areas  for 
consideration  by  the  VV&A  community  in  expanding 
VVML/VDT  utility. 

3.  Revolution:  The  Utility  of  WML  and 
the  VDT 

WML  and  its  application,  the  VDT,  provide  the 
standardization  and  flexibility  needed  to  create 
consistency  in  VV&A  documentation  by  leveraging 
XML  and  providing  users  with  a  physical  and  tangible 
object  to  guide  them  through  the  VV&A 
documentation  process.  WML  creates  consistency 
with  Navy  mandate  by  paralleling  its  contextual 
structure  to  The  Department  of  the  Navy  Modeling  and 
Simulation  Verification,  Validation,  and  Accreditation 
Implementation  Handbook,  Volume  I:  VV&A 
Framework  [7]  while  the  VDT  creates  VV&A 
metadata  files  to  record  the  VV&A  process  and 
progress.  For  each  M&S  project  a  single  metadata  file 
is  created  that  contains  four  related  reports:  the 
Accreditation  Plan  and  Report,  V&V  Plan  and  V&V 
Report.  The  resulting  WML  file  establishes  a 
searchable  index  that  facilitates  configuration 
management,  traceability  and  a  host  of  other  benefits 
that  will  be  addressed  in  the  body  of  this  paper.  The 
flexibility  of  this  web-based  language  and  tool  will 
also  save  the  Navy  valuable  time  and  money  by 
uniformly  delivering  M&S  projects  for  accreditation 
and  creating  M&S  repositories  with  searchable  entities 
for  future  reference. 


3.1  VDT:  the  VV&A  Implemented’  Interface  with 
WML 

VV&A  implementers  require  a  quick  and  efficacious 
means  to  understand  and  perform  the  VV&A  process. 
The  VDT  fulfills  this  need  by  formatting  the  required 
reports  directly  from  the  VV&A  template  and  creating 
a  manageable  VV&A  task  list.  Correlation  with  the 
VV&A  handbook  [8]  and  SECNAVINST  5200.40A 
[9]  provide  additional  clarification  of  those  sections 
that  new  VV&A  implementers  may  not  understand 
and  facilitates  the  easy  creation  of  required  documents 
and  reports.  Once  the  VV&A  agent  familiarizes 
him/herself  with  the  VDT  template  fields  and  creates  a 
VV&A  task  list,  the  WML  supports  the  execution  of 
these  tasks  by  connecting  all  members  involved  in  the 
M&S  development  and  VV&A  effort  through  a  single 
collaborative  documentation  standard.  The  VDT 
captures  the  textual  documentation  within  specific 
contexts  defined  by  the  WML.  The  collection  of 
WML  files  is  then  stored  in  a  single  repository,  such 
as  the  NMSRR,  creating  a  valuable  asset  to  the  DON 
M&S  communities. 

This  standardization  of  process  inherently  makes 
systems  development  more  formalized  and  rigorous. 
WML  adheres  to  the  standard  the  VV&A  template 
for  documentation  by  representing  the  required  M&S 
VV&A  information  while  the  VDT  uses  WML  to 
surround  the  VV&A  content.  With  the  VDT  portal, 
VV&A  metadata  can  be  captured  for  multiple  projects 
and  will  be  able  to  be  linked  with  the  Navy’s 
NMSRR.  Thus,  WML  promotes  project  visibility 
and  facilitates  information  exchange.  This  specialized 
portal  and/or  the  NMSRR  will  provide  a  robust  web 
interface  for  searching  M&S  projects  using  specific 
VV&A  criteria. 

4.  Evolution:  The  Creation  and 
Development  of  WML  and  the  VDT 

The  VDT’s  first  iteration  was  as  a  stand-alone 
Microsoft  Access  database  and  has  evolved  into  the 
DON  VV&A  Documentation  Tool,  Versions  1  and  2. 
Developed  by  SPAWAR  Systems  Center,  Charleston, 
multiple  users  tested  Version  1;  their  comments  were 
incorporated  in  the  final  Version  1.  With  the  release 
of  Version  1,  numerous  other  comments  called  for 
collaboration  and  information- sharing  utilities.  The 
success  of  VV&A  hinges  upon  participation  from 
various  members  within  the  program  and 
organizational  resources,  both  internal  and  external. 
Hence,  responses  to  the  VDT  v.l’s  utility  and 
capabilities  were  addressed  to  several  users  including: 
OPNAV,  Marine  Corps,  Navy  and  Army  threat 
systems  offices  and  other  organizations. 

Strong  consideration  was  also  given  to  interoperability 
with  the  NMSRR.  As  the  NMSRR  was  created  to 
promote  M&S  reuse  by  providing  a  centralized 
collection  point  where  a  registered  user  can  enter 


information  about  their  M&S,  the  linking  and 
portability  of  XML  can  allow  certain  fields  of  the 
VDT  to  be  directly  linked  to  NMSRR  and  populating 
the  database  with  current  information  with  the  consent 
of  the  users.  The  automation  of  this  VDT/NMSRR 
posting  will  relieve  M&S  VV&A  agents  from  having 
to  manually  duplicate  the  entry  of  this  vital 
information.  Running  historical  information  that  other 
users  can  access  in  the  NMSRR  facilitates  the 
evaluation  of  existing  M&S  for  reuse  based  on  the 
VV&A  documentation. 

4.1  WML:  Metadata 

Metadata  can  be  stored  as  data;  as  well  as  a  resource 
indicated  as  a  URL,  i.e. 

http://www.navmsmo.navy.milA  one  resource  may 
therefore 


another  resource.  One  goal  of  writing  structured 
Metadata  is  to  include  as  much  of  the  syntax  and 
semantics  such  that  maximum  information  can  be 
acquired  when  referencing  a  Metadata  document.  In 
the  context  of  the  NMSRR,  structured  content  in  the 
metadata  is  paramount  to  populating  the  database  with 
useful  information. 

To  make  this  XML  data  interchange  a  reality,  the 
establishment  of  a  metadata  standard  is  necessary  to 
so  that  its  content  conforms  to  the  Navy’s  VV&A 
policy  and  complies  with  the  World  Wide  Web 
Consortium  (W3C)  XML  Schema  [10].  As  the 
primary  goal  of  the  WML  is  inherently  to  provide  a 
framework  for  tracking  V&V  activity,  the  source  code 
in  Figure  1  is  provided  to  show  the  attribute  tags  used 
to  track  events: 


contain  information  about  itself  and/or _ 

<?xml  version="1.0"  encoding="UTF-8"  ?> 

<! —  edited  with  XMLSPY  v5  rel.  4  U  (http://www.xmlspy.com)  by 
David  Broyles  (Home)  — > 

-  <xs:  schema  xmlns: xs="http://myw.w3.org/2001/XMLSchema"> 

-  <xs: element  name="ACC_ACTIVITIES"> 

+  <xs:complexType> 

-  <xs:  keyref  name="ACC_ACTIVITIES_VV_AND_A" 

re  fer= "  V  V_  A  N  D_  A_M  A I  ISI_  V  V_  A  N  D_  A_I  D "  > 

<xs: selector  xpath="."  /> 

<xs: field  xpath="VV_AND_A"  /> 

</xs: keyref> 

</xs:element> 

-  <xs:element  name=  "ACC_EVEI\ITS  "> 

-  <xs:complexType> 

-  <xs:sequence> 

<xs:element  name="ACC_EVEISITID"  type="xs:integer" /> 

<xs: element  name="VV_AND_A"  type="xs:integer"  /> 

-  <xs: element  name="COMPLETIOIM_DATE"> 

-  <xs:simpleType> 

-  <xs: restriction  base="xs:string"> 

<xs:maxLength  value="10"  /> 

</xs:restriction> 

</xs:simpleType> 

</xs:  element> 

<xs: element  name="RESPONSIBLE_PARTY" 
type="xs:integer"  /> 

<xs: element  name="EVEIMT_TYPE"  type="xs:integer"  /> 
</xs:sequence> 

_ </xs:  comolexT  vpe> _ 


Figure  1-  XML  Code  segment 


4.2  WML:  Hyperlinks  and  XML  Sub-languages 

Further  supporting  the  use  of  XML  and  web-based 
applications  as  the  basis  for  VV&A  documentation 
standardization  are  hyperlinks,  which  provide  a 
simple,  yet  powerful,  means  of  linking  documents.  As 
such,  WML  applies  the  the  XML  Linking  Language 
(XLink),  XInclude  and  XPointer  languages  to 
facilitate  the  WML’s  role  of  Metadata  Collector  and 


future  capabilities  discussed  later  in  this  document. 
Because  XLink  allows  more  abstract  linking  than 
standard  HTML,  users  can  insert  and  move  elements 
into  XML  documents  with  greater  flexibility,  the 
creation  and  description  of  links  between  resources 
becomes  more  robust.  “XLink  provides  a  framework 
for  creating  both  basic  unidirectional  links  and  more 
complex  linking  structures^]  allowing]  XML 
documents  to  assert  linking  relationships  among  more 


than  two  sources,  associate  Metadata  with  a  link,  and 
express  links  that  reside  in  a  location  separate  from 
the  linked  resources  [11].”  By  using  the  XML  syntax 
to  create  structures  that  can  describe  both  simple, 
unidirectional  hyperlinks  (similar  to  HTML)  and  more 
sophisticated  links,  XLink's  value  to  WML  and  the 
VV&A  practitioner  is  that  it  can  include  a  wide  array 
of  documents.  Audio,  video,  database  data,  training 
media,  schedules,  and  even  XML-enabled  test  results 
can  thus  be  added  to  any  VV&A  file.  This  will  be 
particularly  useful  in  the  realm  of  regression  testing  by 
linking  statistical  applications  directly  to  the  VDT. 

While  XLink  is  an  XML-based  language  that  specifies 
constructs  for  advancing  linking  in  XML  documents, 
XPointer  is  a  non-XML  language  that  allows  the  user 
to  address  the  internal  structures  of  XML  documents 
by  specifying  link  sources  and  targets  [12],  The 
XPointer  language  can  be  used  as  the  basis  as  a 
fragment  identifier  for  any  Uniform  Resource 
Identifier  (URI)  reference.  By  definition,  a  URI  is  a 
series  of  characters  used  to  differentiate  names.  URIs 
locate  a  resource  whose  Internet  media  type  is  one  of 
text/xml,  application/xml,  text/xml-external-parsed- 
entity,  or  application/xml-external-parsed-entity  [13]. 
The  XPointer’ s  capacity  to  harness  URI  fragments 
allows  the  VDT  to  manage  validation  test  parameters. 
In  contrast,  XLink  allows  the  management  of 
parameters  such  parameters  as  schedules  and  costs  is 
shown  in  Figure  03  below  as  a  requirements  matrix 
that  links  pre-test  predictions  to  test  results.  Using  the 
XPointer  specification,  the  VDT  can  reference  test 
results  pointing  directly  to  another  application  by 
simply  pointing  to  the  URI  of  the  application. 

Finally,  the  WML  incorporates  the  XInclude 
specification  to  provide  the  mechanism  for  organizing 
and  capturing  non-text  data  such  as  graphics, 
embedded  objects  or  files.  XInclude  provides  a  link 
with  the  attribute  value  of  embedded  resources  in 
documents  generated  by  the  VDT.  Such  links  provide 
syntax  independent  of  media  type  to  indicate  that  a 
resource  is  to  be  embedded  graphically  within  the 
display  of  the  document  [14].  This  capability  has  far- 
reaching  implications.  For  example,  it  can  add  the 
ability  to  link  live  schedule  information  or  even  model 
information  directly  from  the  M&S  applications  to  the 
VV&A  project  file. 

5.  UTILITY:  The  Benefits  &  Current 
Limitations  of  the  VDT/VVML 

The  benefits  of  WML  and  the  VDT  are  numerous: 
from  one-step  standardization  and  aggregation  to 
universal  applicability.  Further  utility  and  expansion 
into  the  area  of  M&S  scheduling  and  management 
covers  only  one  feature  that  the  VDT  may  hope  to 
achieve  in  the  future.  However,  it  is  also  important  to 
note  the  current  limitations  of  WML  and  the  VDT, 
not  only  its  benefits.  As  such,  this  section  seeks  to 


provide  the  road  ahead  as  well  as  the  current  status  of 
WML  and  the  VDT’s  utility. 

5.1  One-Step  Standardization  and  Aggregation 

The  main  benefit  of  using  the  VDT  over  Microsoft 
Word®  is  that  only  the  VDT  facilitates  standardized 
preparation  of  VV&A  documentation  and  supports  the 
aggregation  of  V&V  information  in  an  M&S 
repositories,  such  as  the  NMSRR.  The  result  is 
similar  to  using  tax-preparation  software  to  fill  in  your 
tax  forms  and  file  your  income  tax  return 
electronically.  Both  the  preparer  and  the  government 
easily  achieve  greater  levels  of  consistency  and 
efficiency  by  using  this  standardized  method. 

The  overall  VDT  process  produces  standardized 
representations  that  can  be  scaled  to  accommodate 
both  small  to  large  M&S  projects.  Specifically, 
WML  is  structured  to  provide  the  four  report 
templates  articulated  in  the  M&S  VV&A 
Implementation  Handbook  Volume  I  [15]  in  electronic 
form: 

•  Accreditation  Plan 

•  Verification  And  Validation  Plan 

•  Accreditation  Report 

•  Verification  And  Validation  Report 

These  four  reports  contain  documentation  of  the 
specific  VV&A  plan  details,  correlating  to  the  VV&A 
process.  These  processes  include: 

•  Data  Verification 

•  Data  Validation 

•  Design  Verification 

•  Implementation  Verification 

•  Conceptual  Model  Validation 

•  Results  Validation 

•  V&V  Reporting 

Thus  the  VDT  and  its  WML  underlay  form  the  only 
tool  that  not  only  creates  standardized  documentation, 
robust  flexibility,  and  XML-based  utility,  but  also 
ensures  compliance  with  Navy  M&S  mandates. 

5.2  EASE  OF  USE  &  UNIVERSAL 
APPLICATION 

As  noted  above,  the  Web-based  VDT  application 
utilizes  XML  to  enable  its  underlying  database  to 
create  an  easily  accessible  user  interface  that  supports 
mandated  Navy  VV&A  standards.  The  VDT  is 
accessible  from  any  browser  and  connects  to  a 
specialized  Navy  website  that  serves  the  latest  VDT 
version.  The  web  interface  supports  multiple 
platforms  to  ease  and  assist  the  user  to  produce 
standardized  Navy  VV&A  documentation.  In  order  to 
consistently  fulfill  this  tall  order;  verify  that  all 
VV&A  standards  are  met;  and  ensure  that  any  user, 
regardless  of  familiarization  with  the  VV&A  process, 
can  easily  accomplish  all  tasks,  the  VDT  represents  a 
tree  of  documentation  headings  with  user  directions 
and  an  edit  pane  that  accepts  formatted  text  and 
pictures,  as  depicted  in  Figure  02: 


Figure  2-  The  VDT  main  window  displays  the  content  associated  with  the  “System/Process  Modeled”  subject  of 
the  V&V Accreditation  Plan. 


Additionally,  the  current  iteration  of  the  VDT 
interface  is  sufficient  to  represent  all  M&S  acquired, 
developed,  managed,  and  used  by  DON  activities.  The 
fact  that  all  categories  of  M&S  used  by  DON 
activities  (including  live,  virtual,  and  constructive 
simulations;  distributed  simulations;  federates  and 
federations;  emulators;  prototypes,  simulators;  and 
stimulators)  can  harness  this  tool  with  relative  ease 
with  little  additional  effort  is  in  itself  a  revolution  in 
M&S  VV&A  documentation.  In  the  increasingly 
demanding  world  of  Navy  M&S  management,  the 
importance  of  the  VDT  for  project  developers,  system 
engineers,  system  architects,  testers,  and  management 
attempting  to  balance  their  traditional  responsibilities, 
challenging  schedules,  and  tight  budgets  with  the  new 
demands  of  M&S  VV&A  can  hardly  be  understated. 


As  discussed  earlier,  the  VDT  creates  each  report 
separately  by  user  request  to  the  format  prescribed  by 
the  Navy  standards  for  VV&A  documentation.  For 
easier  access  and  standard  formatting,  project  VV&A 
content  is  sequentially  placed  between  indexed 
documentation  headers  while  maintaining  the  user- 
supplied  formatting  intact.  Each  report  is  then 
generated  and  can  be  previewed  using  the  VDT  using 
the  universally  recognizable  “Print  Preview”  menu 
selection.  The  “Print”  menu  selection  generates  and 
sends  a  standardized  report  to  the  specified  printer. 
The  following  is  an  example  of  the  Print  Preview 
window: 


Figure  3-  Standard  Print  Preview  window 


5.3  Current  VDT  Limitations 

The  VDT  directly  manages  the  WML  structure  and 
document  content  to  the  new  VV&A  standard. 
Because  the  VDT  can  include  “cut-and-paste”  data 
from  other  editors  and  tools,  the  VDT  inspires 
thoughts  of  composability  and  substantive 
interoperability  between  M&S  components.  The  VDT 
can  represent  and  relates  a  user’s  high-level 
perspectives  of  the  usefulness  of  M&S.  However, 
further  development  is  needed  for  the  VDT  to  provide 
users  with  these  advanced  features  and  capabilities. 
Some  known  issues  include: 

•  Dependent  on  Web  connectivity 

•  The  addition  of  a  spell  check  feature 

•  The  capacity  to  add  a  table  of  contents,  page 
numbers,  and  indexes 

•  Binary  representation  of  images  that  occupy 
less  disk  space  that  XML-wrapped  rich  text 

5.4  The  Tailoring  of  V&V 

The  process  of  VV&A  can  be  exhaustive  and  costly, 
so  it  is  suggested  that  this  judgment  process  be 
conducted  using  common  sense,  sound  business 
management  practices,  and  considerate  of  time  and 
resources  efficiencies.  We  believe  that  organizations 
can  adhere  and  comply  with  regulations  to  do  VV&A, 
follow  the  guidelines,  and  produce  meaningful  VV&A 
documentation  efficiently  with  the  use  of  the  WML 
Documentation  Tool.  Documentation  content  is  the 
sole  responsibility  of  VV&A  practitioners.  As  such, 
portions  of  the  VV&A  content  may  be  left  blank, 
resulting  in  abridge  VV&A  reports. 

6.  Further  Fruition:  Future  Capabilities 

6.1  Flexibility  and  the  Additional  Functionality 

In  2004  the  release  of  a  new  standard  for  Navy  VV&A 
documentation  clarifies,  characterizes,  and  evolves  the 
VV&A  process.  In  synergy  with  this  evolution, 
WML  emerges  as  the  single  standard  to  represent 
project  VV&A  artifacts.  Because  WML  is  a 
language  that  can  easily  adapt  to  future  requirements 
and  evolving  VV&A  methods,  software  development, 
and  VV&A  process  maturity,  higher  levels  of 
validation  categorization  and  new  means  of 
requirements  traceability  can  be  established  and 
standardized  through  the  VDT.  WML  also  has  the 
capacity  to  adapt  to  additional  XML  tags.  WML 
extensions  open  new  possibilities  to  represent 
complex  relationships  between  M&S  components. 
Often  expensive,  proprietary  tools  appear  to  fill  the 
current  void.  However,  continued  improvements  to 
the  VDT  will  extend  and  improve  the  management  of 
the  VV&A  process,  including: 

•  Monitors  and  metrics  of  VV&A  activities 
and  progress 


•  The  provision  of  essential  feedback  regarding 
the  organization  of  VV&A  activities  with 
M&S  development  lifecycle. 

•  The  collection  and  summarization  of  lessons 
learned  to  extrapolate  possible  changes  and 
improvements  to  future  VV&A  processes 
and  standards 

•  The  ability  to  expand  data  import  and  export 
methods  including  requirements  traceability 
matrixes  (RTM)  and  other  tool  matrixes. 

6.2  VDT  Support  of  DON  Web-Centricity  and 
Streamlined  Schedules 

With  regard  to  the  DON’S  emerging  web-centric 
approach,  the  VDT  application  has  the  possibility  of 
improving  the  management  of  VV&A  processes  by 
providing  timely  visibility  of  progress,  audits, 
summary  analysis,  and  judgments  of  credibility.  By 
using  the  VDT,  M&S  managers  can  coordinate  their 
efforts  to  respond  to  critical  issues  of  performance  and 
effectiveness,  thereby  streamlining  the  effort  to 
quantify  and  mitigate  risks.  Future  versions  of  the 
VDT  will  manage  a  template  that  extends  for  each  of 
the  project  M&S  components.  After  the  VV&A 
process  is  tailored  to  fit  the  project,  a  schedule  can  be 
created  that  associates  VV&A  activities  and  time  with 
costs.  Thus,  the  VDT  has  the  potential  to  quantify  and 
track  costs.  This  functionality  will  help  evaluate 
project  risk  as  a  relationship  of  historical  costs, 
budgeted  costs,  and  projected  costs. 

As  the  VV&A  Documentation  Tool  suggests  a 
roadmap  for  the  direction  and  completeness  of  the 
documentation  based  on  the  stage  of  the  M&S 
development  cycle,  the  VDT  can  evolve  to  promote 
each  specific  V&V  process  M&S  development  stage. 
The  V&V  calendar  can  be  used  as  a  simplified 
resource  planner  to  organize  high-level  V&V  tasks. 
As  a  Web-accessible  tool,  the  VDT  can  support  a 
standardized  process  to  efficiently  capture  and  share 
V&V  data  for  streamlined  documentation  and 
collaboration  while  V&V  activities  are  audited 
throughout  the  execution.  Hence,  the  VDT  emerges 
from  a  documentation  template  program  to  a  full 
service  VV&A  assistant  that  supports  an 
organization's  advance  into  process  maturity. 

The  VDT  organizes  VV&A  information  that  identifies 
how  the  M&S  project  can  sufficiently  execute  VV&A 
and  meet  acceptability  criteria.  When  actual  results  of 
V&V  testing  are  available,  M&S  management  gains  a 
cohesive  understanding  of  the  M&S  progress  by 
tapping  into  this  rich  resource,  thus  casting  away  the 
vagueness  and  opacity  so  often  noted  as  a  frustration 
of  VV&A.  With  such  openness  and  clarity,  the 
acceleration  of  the  overall  M&S  schedule  is  likely  as 
developers  will  be  able  to  identify  and  focus  on 
higher-priority  tasks  which  directly  relate  to  the  most 
important  M&S  acceptability  criteria.  Because  WML 
has  the  capability  to  conjoin  system-level  tests  with 


expected  results,  the  applications  of  WML  in  the 
VDT  can  be  expanded  ad  infinitum  to  create  and 
maintain  a  matrix  of  relationships  that  represent  the 
acceptability  criteria  and  the  status  of  system-level 
tests  and  analysis. 

6.3  Composability 

The  capacity  for  any  one  standardized  template  to  be 
deployed  without  knowledge  of  the  M&S  developer’s 
methodology  (object-oriented,  functional 

decomposition,  or  fast-prototyping),  has  been  noted 
during  the  VDT  development  effort.  By  encapsulating 
the  details  WML  uses  to  represent  a  structure  of  data, 
models,  and  simulation  component  documentation  in 
the  extensive  template  framework,  documentation 
interoperability  can  be  substantively  evolved  via 
WML  generation  and  links.  WML  supports  true 
composability  among  M&S  iteratively  feeding  the 
VDT  new  tools.  This  iterative  process  also  achieves 
technical  interoperability. 

A  concerted  effort  by  the  SISO  Extensible  Modeling 
and  Simulation  Framework  (XMSF)  Group  [16] 
promotes  the  concept  of  composable  components 
using  the).  In  order  to  arrange  and  compose  new 
M&S  projects  and  configurations,  M&S  will  require 
consistent  and  standardized  nomenclature,  high-level 
descriptions,  and  the  designation  of  each  M&S’s 
specific  solution.  It  is  important  to  note  that  although 
WMF  is  designed  to  support  associations  in  the 
Navy’s  standard  of  the  VV&A  documentation,  certain 
sections  of  VV&A  documentation  describe  unique 
qualities:  system  components  and  their  associated 
requirements,  acceptability  criteria,  assumptions, 
qualifications,  constraints,  and  test  results.  Those  who 
would  compose  an  M&S  need  a  strong  understanding 
of  the  various  implementation  techniques  and  their 
purposes.  In  such  circumstances,  there  may  be  no 
better  representation  of  the  documented,  analyzed,  and 
approved  M&S,  than  the  project’s  WML  file. 

6.4  M&S  Reuse  and  Cost  Savings 

The  consistent  use  of  the  VDT  and  WMF  directly 
aids  in  effort  to  reuse  M&S  by  standardizing  original 
VV&A  documentation  for  ready  accessible  for 
reference,  reducing  the  cost  of  the  overall  costs  of 
legacy  VV&A.  VV&A  documentation  provides  a 
concrete,  value-added  deliverable  for  that  benefits  the 
entire  M&S  community.  Developers  who  evaluate 
M&S  for  reuse  will  not  only  have  easily  accessible 
and  consistent  VV&A  documentation,  but  also  access 
to  the  original  validation  test  results  and  expected 
outcomes  in  one  file,  thus  streamlining  the  future 
efforts  to  understanding  of  the  legacy’s  M&S 
performance  and  capabilities. 

7.  Summary 


The  Navy  Simulation  Baseline  Assessment  [17]  for 
FY02  reported  a  DON-wide  expenditure  on  M&S  of 
over  $380M  in  the  areas  of  training  and  education, 
operations  and  experimentations,  acquisitions, 
assessment,  and  management,  science,  and 
technology.  With  so  much  riding  on  the  line,  the 
standardization  of  VV&A  is  no  longer  a  nicety;  it  is  a 
necessity.  WMF  and  its  implementation  via  the  VDT 
specifically  address  this  exigency  by  developing  a 
single  Navy  VV&A  documentation  standard. 

Together,  WML  and  the  VDT  streamline  VV&A  by 
automatically  generating  standardized  documentation 
and  saving  the  information  as  constant  WML  files. 
WML  provides  a  specialized  XML-based  language 
to  create  a  unique  Navy  VV&A  documentation 
template.  The  VDT  dynamically  applies  WML  to 
aggregate  the  VV&A  content.  It  is  the  VDT’s  use  of 
WML  that  solidifies  a  solution  that  aggregates  the 
VV&A  documentation  process  and  provides  a  more 
meaningful  representation  of  VV&A  information.  As 
the  foundation  of  WML  is  the  highly  malleable 
XML,  future  growth  in  addressing  the  growing 
demands  of  M&S  VV&A,  its  management,  and 
processes  and  flexibility  is  virtually  limitless.  WML, 
the  foundation  of  the  VDT,  may  lead  to  advancements 
in  how  VV&A  results  are  composed,  organized, 
distributed,  searched,  shared,  and  reused. 
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9.  Acronyms 

DII  COE  Defense  Information  Infrastructure 
Common  Operating  Environnment 
DON  Department  of  the  Navy 

HLA  High  Level  Architecture 

HTML  Hyper  Text  Markup  Language 

M&S  Modeling  and  Simulation 

NAVMSMONavy  Modeling  and  Simulation 
Management  Office 

NMSRR  Navy  Modeling  and  Simulation 
Resource  Repository 

OPNAV  Office  of  the  Chief  of  Naval  Operations 
SECNAVINST  Secretary  of  the  Navy  Instruction 
T&E  Test  and  Evaluation 

URI  Uniform  Resource  Identifier 

URL  Uniform  Resource  Locator 

VDT  DON  Verification,  Validation,  and 

Accreditation  Documentation  Tool 
VV&A  Verification,  Validation,  and 
Accreditation 

WML  Verification,  Validation,  and  Accreditation 
Markup  Language 

W3C  World  Wide  Web  Consortium 

XML  Extensible  Markup  Language 

XMSF  Extensible  Modeling  and  Simulation 

Framework 
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